chore(ci): fix Operator tests, artifact handling, pod selectors - #4414

Description

wait_for_crunchy_crd was extracted from deploy_rhdh_operator, but the OCP operator job still deploys
the base and runtime Backstage CRs without calling it. On OpenShift (where the code comment says
CrunchyDB is required), this can race the PostgresCluster CRD and cause the operator’s internal DB
provisioning to fail intermittently.

Code

.ci/pipelines/install-methods/operator.sh[R54-70]

+wait_for_crunchy_crd() {
  # Ensure PostgresCluster CRD is available before deploying Backstage CR
-  # This is required because the operator relies on CrunchyDB for its internal database
+  # This is required because the operator relies on CrunchyDB for its internal database on OpenShift
  log::info "Verifying PostgresCluster CRD is available before deploying Backstage CR..."
  k8s_wait::crd "postgresclusters.postgres-operator.crunchydata.com" 60 5 || {
    log::error "PostgresCluster CRD not available - operator won't be able to create internal database"
    return 1
  }
+}
+
+deploy_rhdh_operator() {
+  local namespace=$1
+  local backstage_crd_path=$2

-  # Verify Backstage CRD is also available
+  # Verify Backstage CRD is available
  k8s_wait::crd "backstages.rhdh.redhat.com" 60 5 || return 1

Evidence

deploy_rhdh_operator no longer checks/blocks on the PostgresCluster CRD, and wait_for_crunchy_crd is
only explicitly invoked in the RBAC deployment path in the OCP operator job; the base and runtime
operator deployments call deploy_rhdh_operator directly. The new wait_for_crunchy_crd comment also
states this CRD is required on OpenShift, so skipping it for base/runtime reintroduces the original
race condition the check was meant to prevent.

.ci/pipelines/install-methods/operator.sh[54-70]
.ci/pipelines/jobs/ocp-operator.sh[16-27]
.ci/pipelines/jobs/ocp-operator.sh[32-41]
.ci/pipelines/jobs/ocp-operator.sh[87-93]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`wait_for_crunchy_crd` was moved out of `deploy_rhdh_operator()`, but OCP operator flows still call `deploy_rhdh_operator` for **base** and **runtime** deployments without waiting for the Crunchy PostgresCluster CRD. This can cause intermittent failures on OpenShift where the operator expects CrunchyDB.

### Issue Context
- `wait_for_crunchy_crd()` exists and is documented as required on OpenShift.
- RBAC operator deployments call it, but base and runtime do not.

### Fix Focus Areas
- Add a guarded Crunchy wait (only when `IS_OPENSHIFT==true`) before applying the Backstage CR for **all** OCP operator deployments, either:
 - inside `deploy_rhdh_operator()` (conditionally), or
 - at each OCP call site (base + RBAC + runtime).

- .ci/pipelines/install-methods/operator.sh[54-97]
- .ci/pipelines/jobs/ocp-operator.sh[16-27]
- .ci/pipelines/jobs/ocp-operator.sh[87-93]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Description

waitForDeploymentReady derives podSelector only from spec.selector.matchLabels and then always calls
logPodConditions(namespace, podSelector), even when podSelector is empty. If a Deployment selector
uses matchExpressions (or otherwise has no matchLabels), this will list/log every pod in the
namespace, slowing and polluting readiness checks.

Code

e2e-tests/playwright/utils/kube-client.ts[R636-655]

+        // Derive the pod label selector from the deployment's matchLabels
+        const matchLabels = response.body.spec?.selector?.matchLabels || {};
+        const podSelector = Object.entries(matchLabels)
+          .map(([k, v]) => `${k}=${v}`)
+          .join(",");
+
        console.log(`Available replicas: ${availableReplicas}`);
        console.log(
          "Deployment conditions:",
          JSON.stringify(conditions, null, 2),
        );

        // Check for pod failure states when expecting replicas > 0
-        if (expectedReplicas > 0) {
+        if (expectedReplicas > 0 && podSelector) {
          const podFailureReason = await this.checkPodFailureStates(
            namespace,
-            labelSelector,
+            podSelector,
          );
          if (podFailureReason) {
            console.error(

Evidence

podSelector is built exclusively from matchLabels and can become an empty string; despite guarding
pod failure checks on podSelector, the code still unconditionally logs pod conditions with that
selector. logPodConditions forwards the selector directly to listNamespacedPod; an empty selector
results in an unfiltered list call (all pods).

e2e-tests/playwright/utils/kube-client.ts[617-667]
e2e-tests/playwright/utils/kube-client.ts[737-810]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`waitForDeploymentReady()` always calls `logPodConditions(namespace, podSelector)` even when `podSelector` is empty (because it is derived only from `spec.selector.matchLabels`). An empty selector causes the pod list call to return **all pods in the namespace**, creating noisy output and potential slowness/timeouts.

### Issue Context
- Some Deployments may use `spec.selector.matchExpressions` without `matchLabels`.
- There is already a safer helper `logPodConditionsForDeployment()` that avoids logging when no selector can be resolved.

### Fix Focus Areas
- In `waitForDeploymentReady()`, only call `logPodConditions` when `podSelector` is non-empty, otherwise:
 - call `logPodConditionsForDeployment(deploymentName, namespace)`, or
 - enhance selector derivation to also support `matchExpressions`.

- e2e-tests/playwright/utils/kube-client.ts[636-667]
- e2e-tests/playwright/utils/kube-client.ts[737-810]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.